Graphical risk-based performance measurement and benchmarking system and method

ABSTRACT

A system and method provides analysis of pre- and post-project effects on frequency and severity of operational loss. The method includes the computation of the frequency of incidents and the severity of incidents for equipment based on recorded incident data. Risk, a function of frequency and severity, is calculated before and after a project to determine if the project has changed the risk of an incident in a piece of equipment or facility and if the project has changed the nature of risk (i.e. increasing severity while decreasing frequency). The method may be used to compute forecast estimates of future operational losses, assess goals for improving current performance relative to demonstrated industry performance, and determine statistical confidence intervals of forecast such that risk, and changes in risk, may be visually quantified and communicated.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 60/961,179, filed Jul. 19, 2007, which is incorporated by referencein its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to a system and method for comparative operationalperformance analysis and benchmarking and in a preferred embodiment tothe analysis and benchmarking of the operational effects of automationutilizing a two-dimensional graphical depiction of risk that visualizeschanges in performance frequency, severity, and risk.

2. Background Summary

In the fields of automation and advanced process controls, technologicaladvances often result in the commercialization of new products that willpresumably provide reliability and production savings. In addition,certain changes in operational procedures, training, and maintenancepractices implemented at industrial facilities can have similarperformance improvement objectives. However, a problem in the artexists, since no systems or methods are known prior to the inventiondisclosed herein, which effectively quantifies performance improvements,such as reliability and production.

In practice, measuring quantitative change in industrial processes iscomplicated by the concomitant effects of external and internal factorsthat are simply part of what is occurring at the facility or facilitiesundergoing the performance improvement activities. Since no generalmethod exists to censor non-relevant factors, determining the effects ofnon-applicable external and internal factors is a challenge. Forexample, data must be screened to only measure improvements on specificequipment, process units, or areas and control groups must beestablished to test for placebo effects.

BRIEF SUMMARY OF THE INVENTION

The concept of risk is applied to address the problem of measuringperformance. Risk, defined as future's uncertainty, can be modeled inindustrial performance settings as the product of process or equipmentincident frequency and incident severity. An incident is a loss eventthat may be a failure event, such as a breakdown, or a non-failureevent, such as a planned shutdown. Risk is an abstract quantity in thesense it is subjectively interpreted and valued differently betweenindividuals or groups. A graphical depiction of the mathematicalcalculation of risk as a function of its two components (incidentfrequency and incident severity) enables the communication of the basicinformation in a visual framework that is more easily understood by awide range of constituents. The visual framework enables readers toeasily identify movement and subsequent changes in frequency, severity,and risk from visual inspection of the graphs. Even though the sameinformation can be obtained from numerical tables, for most people,reviewing a visual plot is easier than studying process changesreflected in columns of numbers.

A system and method quantifies changes in an industrial process,equipment reliability, or in any entity that is subject to productionlosses that can be defined through the number of loss events and theircorresponding severity. One embodiment provides a system and method toquantify the effects of automation and communicate changes in anindustrial process, equipment reliability, etc. Loss events may becensored to exclude losses that are desired to be excluded from theanalysis for several reasons, e.g., the losses are outside the timeperiods of interest, or are not pertinent to performance improvement,such as weather, economic conditions, etc. The data may then be dividedinto a baseline, automation pre-installation or pre-improvement projecttime period and an automation post-installation period. The data may befurther divided into sub-periods such as years, quarters, or months. Thedata may then be applied to compute incident frequency defined as thenumber of incidents in each time period divided by the time periodduration and incident severity defined as the average loss per incident.

One embodiment is a computer-implemented method for computing the riskof pre- and post-project incidents and communicating thefrequency-severity of the incident data comprising the steps of:collecting incident data; computing total loss for pre- and post-projectincidents; computing the frequency of the pre- and post-projectincidents; computing the severity of the pre- and post-projectincidents; computing the risk of the pre- and post-project incidentsusing the frequency and severity of the pre- and post-project incidents;and generating a frequency-severity framework with iso-risk curves. Afurther extension of this embodiment may include the step of rendering,e.g., displaying on a computer monitor, printing, plotting, etc.,frequency, severity, and risk incident data.

Another embodiment is a method for providing analysis of pre- andpost-project effects on operation loss frequency and severity comprisingthe steps of: collecting incident data; computing total loss for pre-and post-project incidents; computing the frequency of the pre- andpost-project incidents; computing the severity of the pre- andpost-project incidents; computing the risk of the pre- and post-projectincidents using the frequency and severity of the pre- and post-projectincidents; computing a forecast estimate for the frequency of incidents;computing a forecast estimate for the severity of incidents; computing aforecast estimate for the risk of incidents based on the forecastestimates for frequency and severity; and generating afrequency-severity framework with iso-risk curves. A further extensionof this embodiment may include the step of rendering thefrequency-severity of the incident data. An even further extension ofthis embodiment may include the steps of rendering and generating thefrequency-severity incident data for at least one forecast estimate onthe visual representation.

Yet another embodiment is a method for providing analysis of pre- andpost-project effects on operation loss frequency and severity comprisingthe steps of: collecting incident data; computing total loss for pre-and post-project incidents using the; computing the frequency of thepre- and post-project incidents; computing the severity of the pre- andpost-project incidents; computing the risk of the pre- and post-projectincidents using the frequency and severity of the pre- and post-projectincidents; computing a forecast estimate for the frequency of incidents;computing a forecast estimate for the severity of incidents; computing aforecast estimate for the risk of incidents based on the forecastestimates for frequency and severity; computing a confidence intervalfor at least one forecast estimate; generating a frequency-severityframework with iso-risk curves. A further extension of this embodimentmay also include the step of generating the frequency-severity of theincidents for at least one forecast estimate. An even further extensionof this embodiment may also include the steps of computing and renderinga confidence interval for at least one forecast estimate.

A system according to one embodiment comprises a server, comprising: aprocessor, and a storage subsystem; a database stored by the storagesubsystem comprising: incident data; a computer program stored by thestorage subsystem, when executed causing the processor to: collectincident data; compute the total loss for pre- and post-projectincidents; compute the frequency of the pre- and post-project incidentscompute the severity of the pre- and post-project incidents; compute therisk of the pre- and post-project incidents using the frequency andseverity of the pre- and post-project incidents; and generating afrequency-severity framework with iso-risk curves. In a furtherextension of this embodiment, when executed, the program causes theprocessor to rendering the frequency, severity, and risk of the incidentdata.

Another embodiment of a system comprises a server, comprising: aprocessor, and a storage subsystem; a database stored by the storagesubsystem comprising: incident data; a computer program stored by thestorage subsystem, when executed causing the processor to: compute thetotal loss for pre- and post-project incidents; compute the frequency ofthe pre- and post-project incidents; compute the severity of the pre-and post-project incidents; compute the risk of the pre- andpost-project incidents using the frequency and severity of the pre- andpost project incidents; compute a forecast estimate for the frequency ofincidents compute a forecast estimate for the severity of incidents;compute a forecast estimate for the risk of incidents using the forecastestimates for frequency and severity; and generating afrequency-severity framework with iso-risk curves. In a furtherextension of this embodiment, when executed, the program causes theprocessor to generate the frequency-severity of the incident data on thevisual representation. In an even further extension of this embodiment,when executed, the program causes the processor to render thefrequency-severity incident data for at least one forecast estimate.

Yet another embodiment of a system comprises a server, comprising: aprocessor, and a storage subsystem; a database stored by the storagesubsystem comprising: incident data; a computer program stored by thestorage subsystem, when executed causing the processor to: compute thetotal loss for pre- and post-project incidents; compute the frequency ofthe pre- and post-project incidents; compute the severity of the pre-and post-project incidents; compute the risk of the pre- andpost-project incidents using the frequency and severity of the pre- andpost-projects incidents; compute a forecast estimate for the frequencyof incidents; compute a forecast estimate for the severity of incidents;compute a forecast estimate for the risk of incidents using the forecastestimates for frequency and severity; compute a confidence interval forat least one forecast estimate; and of a frequency-severity frameworkwith iso-risk curves; and render the frequency-severity of the incidentdata. In a further extension of this embodiment, when executed, theprogram causes the processor to render frequency-severity for at leastone forecast estimate. In an even further extension of this embodiment,when executed, the program causes the processor to render a confidenceinterval for at least one forecast estimate.

The pre- and post-installation points may be plotted with incidentfrequency along the x-axis and incident severity along the y-axis.Iso-risk contours may be added that are in the range of the loss data toshow how risk (frequency*severity) has changed between thepre-installation or baseline period and the post-installation periods.The iso-risk contours are straight lines, if frequency and severity areplotted using logarithmic scaling. Along these lines, the loss which iscalculated as frequency*severity equals the same value.

Iso-risk quartiles can also be added to signify industry performancemetrics to further provide a benchmarking measurement along with theperformance change quantification observed from baseline(pre-installation) to post-installation. These industry performancemetrics may be derived from industry comparative performance analysis ofthe data from industry-wide surveys or may be based on specificbenchmarks.

The steps in the methods and systems disclosed and claimed herein, asapplicable, can be performed by a single entity or multiple entities, ona single system or multiple system, and any or all of the method stepsor system elements may be performed or located in the United States orabroad, all permutations of which are expressly within the scope of theclaims and disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

These and further features will be apparent with reference to thefollowing description and drawings, wherein:

FIG. 1 a is the first part of a flow chart illustrating the analysismethod for comparing pre- and post-project effects on frequency andseverity of operational loss;

FIG. 1 b is the second part of a flow chart illustrating the analysismethod for comparing pre- and post-project effects on frequency andseverity of operational loss;

FIG. 2 is an exemplary frequency-severity framework with the iso-riskcurves for graphing incident data;

FIG. 3 is an exemplary frequency-severity graph with pre- andpost-installation incident data mapped into applicable groups,indicators showing shifts in frequency-severity values for the groups,and divided into risk quartiles;

FIG. 4 is an exemplary frequency-severity graph with pre- andpost-installation incident data mapped into applicable groups andfurther divided by relevant time periods. Frequency-severity shiftindicators are displayed along with forecast estimate data;

FIG. 5 is an exemplary volatility analysis graph showing the confidenceintervals for pre-installation and a post-installation forecast estimateto indicate the statistical significance of an improvement to theproject analyzed; and

FIG. 6 is a diagram of a preferred embodiment of the system that enablesthe analysis method for comparing pre- and post-project effects onfrequency and severity of operational loss.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

In FIGS. 1 a and 1 b, the Graphical Risk-Based Performance Measurementand Benchmarking System according to a preferred embodiment are shown.Incident data 1000 that contains information on the data or time ofloss, the type of loss, and the amount of loss measured in dollars orother units (such as equivalent distillation capacity in the refiningindustry, or as percent of capacity in any manufacturing facility) iscompiled into a pre-analysis file. This compilation then undergoes adata validation process that ensures accurate information is beingentered into the System.

The dataset is then censored 1100 to only include incidents that occurduring specific, pre-determined time periods. The first time period iscalled the “baseline” or “pre-installation” time interval. This refersthe interval of time that performance will be measured a prioriautomation improvements as a benchmark to measure changes. Thesubsequent time period is called “post-project” or “post-installation.These time intervals may be a cumulative period over successive previouspost-project periods. The intent of step 1100 is to remove data thatdoes not fit into the analysis time periods. The data is not split intothe various time periods until step 1500.

Once the data reaches this quality level, statistical/analyticaltechniques 1200 may be applied to the data. For example, statisticalprocedures may be applied to aggregate multiple incident severities thatoccur from the same cause event or severity value may be transformed tofinancial of other user-defined loss value units.

The dataset is now censored 1300 to remove incident causes that are notapplicable to measure or benchmark performance changes between thepreviously specified time periods. For example, losses orprocess/production reductions from normal levels caused by weather,planned maintenance, economic conditions, or other selected causes maybe removed from the analysis at this point.

The data records are classified via a mapping 1400 into categories thatare relevant to the benchmarking and performance measurement studyobjectives. For example, incidents for pumps, fans, and motors might beclassified together as a category called “Hardware/Mechanical.” Thiscategory mapping also enables the tracking of performance for groups ofincidents that would otherwise possess an insufficient number ofincidents to yield meaningful statistical results if analyzedseparately.

At point 1500, the analysis is divided into two sections. The data arepartitioned into two datasets that represent the pre-installation (orpre-project) and post-installation (or post-project) time periods.

The pre-installation data are now analyzed 1600 to compute incidentfrequency for each selected incident category and for the overalldataset. Incident frequency is defined as the quotient of the number ofincidents in the time period and the length of the time period, forexample, measured in days. Incident severity can be defined in severalways. In this embodiment, incident severity is defined as the averageloss per incident. The risk values are computed as the product ofincident frequency and incident severity for each category and for theoverall dataset.

The post-installation data are now analyzed 1650 to compute incidentfrequency for each incident category and for the overall dataset. Thisis done for the overall post installation period and may be done forinterim time periods such as each quarter of the year if this additionallevel of benchmarking detail is desired. The same definitions anddefinition options as in 1600 are applied to computing post-installationincident frequency and incident severity. The post-installation riskvalues are computed as the product of incident frequency and incidentseverity for each category and for the overall dataset.

The post-installation period data may be analyzed using advancedprediction and forecasting methods 1700 to estimate future incidentfrequency, severity and risk by category and/or overall. These methodsmay apply the censored, validated data and other external data asnecessary depending on the technique applied.

In one embodiment, frequency and severity event data feed to the systemin a real time or batch automated mode from instrumentation via cable orwireless to allow the risk plots to represent current operational riskand to identify trends in current operations that can be used to takeactions to prevent unplanned downtime, system/component incidents, orinefficient operations.

The pre-installation and post-installation frequency, severity and riskvalues are applied to scale the risk-based performance measurement andbenchmarking framework 1800. At this step, the scale value ranges aredetermined for the horizontal and vertical axes and the applicableiso-risk contours are drawn. These contours are diagonal lines iffrequency and severity are plotted using logarithmic scaling. Alongthese lines the product of frequency and severity is a constant. Oneanalysis task of this step is to determine which iso-risk lines to showthat are applicable to measure risk changes in the categories andoverall between the pre-installation and post-installation periods. Anexample of this risk-based framework is presented in FIG. 2.

Up to this point only data specifically from the plant or process unitwas utilized in the analysis. At this point 1900, industry comparativeperformance data are applied to compute industry-level values of overallrisk that are pertinent to the benchmarking and performance measurementactivity that are being analyzed. The industry risk values are computedfrom industry comparative performance data and are applied to the riskframework, in a preferred embodiment as performance quartiles asillustrated in FIG. 3. The industry quartiles provide additionalvaluable information to compare the pre- and post-project reliability tothe demonstrated industry-achieved reliability rates. The preferredanalysis method is to compare the overall unit risk to the firstquartile average risk. This comparison reveals if additional room forimprovement exists when comparing the unit's performance to the industryleaders or otherwise to a specific set of benchmark values.

The pre-installation and post-installation frequency and severity pointsare plotted in the risk-based framework 2000 to show the quantitativeand directional changes in these quantities and the correspondingchanges in risk. Arrows can be drawn for each category and the overallvalues between the pre-installation and post-installation values tohighlight the directional aspects of the quantitative changes inperformance. Forecasts for post-installation frequency and severitypoints are plotted in the risk-based framework 2100 to display thepredicted quantitative and directional change. FIG. 3 is an example ofthese results.

The predictive analysis results developed in 1700 can be plotted in thisrisk-based framework as a forecast estimate of frequency, severity, andrisk. FIG. 4 is an example of these results.

For newly constructed facilities, this method can be altered to excludethe baseline data. In this case the industry quartiles in step 1900 areused as the baseline for comparison of results as shown in step 2200.The project is first targeted to achieve a selected risk level as areplacement for step 1600. In a preferred embodiment, the first quartileaverage frequency severity performance is selected as the goal.Post-project data is then compared to first quartile average performanceto determine if the project goals have been met.

Confidence intervals on the forecast values can also be added dependingon the analysis objectives and the applied forecast methods employed instep 2300. These confidence intervals can be applied to the frequency,severity values or other statistics that are derived from the data. Forexample, 95% confidence intervals can be placed on the mean values ofboth or either axis to measure changes using generally acceptedstatistical practices for identifying changes. In another example, shownin FIG. 5, the severity standard deviation is tested for change using95% confidence intervals. This framework visually shows that thestandard deviation has changed in a statistically significant mannerbetween the pre- to post-project time periods. Similar measurements canalso be used to show changes in the average severity, coefficient ofvariation and other statistics.

As shown in FIG. 6, one embodiment of system used to perform the methodincludes a computing system. The hardware consists of a processor 610that contains adequate system memory 620 to perform the requirednumerical computations. The processor 610 executes a computer programresiding in system memory 620 to perform the method. Video and storagecontrollers 630 are required to enable the operation of display 640. Thesystem includes various data storage devices for data input includingfloppy disk units 650, internal/external disk drives 660, internalCD/DVDs 670, tape units 680, and other types of electronic storage media690. The aforementioned data storage devices are illustrative andexemplary only. These storage media are used to enter the incidentfrequency and loss data to the system, store the numerical risk results,store the calculations, and store the system-produced frequency-severitygraphs. The calculations can apply statistical software packages or canbe performed from the data entered in spreadsheet formats usingMicrosoft Excel, for example. The risk calculations are performed usingeither customized software programs designed for company-specific systemimplementations or by using commercially available software that iscompatible with Excel or other database and spreadsheet programs. Thesystem can also interface with proprietary or public external storagemedia 700 to link with other databases to provide industry-levelfrequency, severity, and/or risk data to be applied to the performancemeasurement benchmarking system and method calculations. The outputdevices can be a telecommunication device 710 to transmit thecalculation worksheets and other system produced graphs and reports viaan intranet or the Internet to management or other personnel, printers720, electronic storage media similar to those mentioned as inputdevices 650, 660, 670, 680, 690 and proprietary storage databases 730.These output devices used herein are illustrative and exemplary only.

The foregoing disclosure and description is illustrative andexplanatory, and various changes in the details of the illustratedsystem and method may be made without departing from the scope of theinvention.

1. A computer-implemented method for computing the risk of pre- andpost-project incidents embodied in a non-transitory computer usablemedium having computer readable program code stored therein, that whenprocessed by a computer processor causes the processor to execute themethod comprising the steps of: collecting incident data; collectingcomparative industry performance data; computing total loss for pre- andpost-project incidents; computing a frequency of the pre- andpost-project incidents; computing a severity of the pre- andpost-project incidents; computing a risk of the pre- and post-projectincidents using the frequency and severity of the pre- and post-projectincidents; and generating of a frequency-severity framework withiso-risk curves, wherein generating a frequency-severity framework withiso-risk curves comprises generating industry performance iso-riskcurves based on comparative industry performance data.
 2. Thecomputer-implemented method of claim 1, further comprising the step of:rendering the frequency, severity, and risk of the incidents.
 3. Thecomputer-implemented method of claim 2, further comprising the steps of:computing a forecast estimate for the frequency based on the incidentdata; computing a forecast estimate for the severity; computing aforecast estimate for the risk; and rendering at least one of thecomputed forecast estimates.
 4. The computer-implemented method of claim3, further comprising the steps of: computing a confidence interval forat least one of the computed forecast estimates; and rendering aconfidence interval for at least one of the computed forecast estimates.5. The computer-implemented method of claim 4, further comprising thesteps of: validating the incident data; censoring the incident data fora time period.
 6. The computer-implemented method of claim 5, whereinrendering the frequency-severity, and risk of the incident comprises thestep of: plotting an indicator of the shift between frequency, severity,and risk of the incidents for a time periods.
 7. Thecomputer-implemented method of claim 1, further comprising the steps of:validating the incident data; and mapping the incident data into atleast one study category.
 8. The computer-implemented method of claim 1,further comprising the steps of: validating the incident data; andcensoring the data to include applicable incidents.
 9. Thecomputer-implemented method of claim 1, further comprising the steps of:validating the incident data; and dividing the incident data intobaseline and post-project groups.
 10. The computer-implemented method ofclaim 1, further comprising the step of: comparing the frequency,severity, and risk with the industry performance iso-risk curves. 11.The computer-implemented method of claim 10, wherein generating industryperformance iso-risk curves includes the step of: setting project goalsbased on the industry performance iso-risk curves.
 12. A systemcomprising: a server, comprising: a processor, a storage subsystem; adatabase stored by the storage subsystem comprising: comparativeindustry performance data; incident data; and a computer program storedby the storage subsystem, when executed causing the processor to:collect incident data; compute the total loss for pre- and post-projectincidents; compute the frequency of the pre- and post-project incidents;compute the severity of the pre- and post-project incidents; compute therisk of the pre- and post-project incidents using the frequency andseverity of the pre- and post-project incidents; and generate industryperformance iso-risk curves based on comparative industry performancedata; construct a visual representation of a frequency-severityframework with iso-risk curves; and render the frequency, severity, andrisk of the validated incident data on the visual representation. 13.The system of claim 12, wherein the computer program, when executed,further causes the processor to: render the frequency, severity, andrisk of the incidents.
 14. The system of claim 12, wherein the computerprogram, when executed, further causes the processor to: compute aforecast estimate for the frequency of the incidents; compute a forecastestimate for the severity of the incidents; compute a forecast estimatefor the risk of the incidents; and render at least one of the computedforecast estimates.
 15. The system of claim 12, wherein the computerprogram, when executed, further causes the processor to: compute aconfidence interval for at least one of the computed forecast estimates;and rendering a confidence interval for at least one of the computedforecast estimates.
 16. The system of claim 12, wherein the computerprogram, when executed, further causes the processor to: censor theincident data to a relevant time periods.
 17. The system of claim 16,wherein the computer program, when executed, further causes theprocessor to: render at least one indicator of the shift betweenfrequency-severity of the validated incidents for the relevant timeperiods.
 18. The system of claim 12, wherein the computer program, whenexecuted, further causes the processor to: validate the incident data.19. The system of claim 12, wherein the computer program, when executed,further causes the processor to: map the incident data into at least onestudy category.
 20. The system of claim 12, wherein the computerprogram, when executed, further censors the data to include applicableincidents.
 21. The system of claim 12, wherein the computer program,when executed, further causes the processor to: divides the incidentdata into baseline and post-project groups.
 22. The system of claim 12,wherein the computer program, when executed, further causes theprocessor to: compare the frequency, severity, and risk values of theincident data to the industry performance iso-risk curves.
 23. Thesystem of claim 12, wherein the computer program, when executed, furthercauses the processor to: allows a user to set project goals based on theindustry performance iso-risk curves.